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ABSTRACT 

This paper reports on the results of a four-year study of a consortium of school districts undergoing systemic 
reform . Results are framed in terms of individual district reform status at the end of the four years in six areas: 
curriculum , parent-community involvement , classroom practice , professional development , administration and 
leadership , and system change. As an overall result of the analysis of these areas districts were identified as low, 
moderate, or high in reform status \ Comparisons were made of reform status and gain in student achievement, 
provision of additional funding (half of the districts received), and demographics of the district. High reform districts 
had larger gains in student achievement but even these gains were less than overall state gains for the same period. 
Funding presented a mixed picture in terms of reform status. Also included is an appraisal of the consortium model 
as a means to provide technical assistance to a group of districts. 

BRIEF BACKGROUND: USE OF A CONSORTIUM MODEL 

The move toward systemic reform in education can be traced to the late 1980’s (McAdams, 
1997, Vinovskis, 1996) when the results from studies of earlier reforms intended to implement a 
more intellectually challenging curriculum and result in improved student results showed few real 
improvements. Policy analysts focused on three areas of concern: incompatible policy signals in the 
system, policies that push toward mediocrity, and the need for district capacity building. Well- 
established student achievement goals are required as well as a review of the resources necessary to 
attain them, especially in high poverty schools (Clune, 1993, O’Day & Smith, 1993). The need for a 
more integrated and comprehensive approach involving multiple components of the educational 
system as well as federal, state, and local policymakers was evident. To accomplish this O’Day and 
Smith (1993) called for strong state and national roles while Clune (1993) suggested alternatives to 
centralization but also acknowledged the need for a systemic educational policy. As often happens, 
a midground has evolved with no national curriculum or national assessments but varying state 
requirements in both areas. The standards-based reform movement gained momentum during the 
same decades and in many states the establishment of state standards became the predominant state 
role and provided direction for schools undergoing systemic reform efforts (Fuhrman, 1994). 

As systemic reform efforts were initiated in districts, studies revealed the lack of teacher and 
administrator capacities to succeed with reform initiatives and suggested a need for eternally 
provided professional development and technical assistance (Massell, Kirst, & Hoppe, 1997; 
Stephens, Leiderman, Wolf, & McCarthy, 1994). Studies of successful districts (Sammons, 1999; 
Shields, Knapp, & Weschler, 1995; St.John & Pratt, 1997) indicated the importance of high 
expectations for student accomplishment, pervasive instructional focus, adequate local leadership, 
supporting relationships with the national reform community, teachers able to use national standards 
to guide instruction, and the building of new local programs such as reform-based professional 
development plans. States themselves scrambled to deliver the assistance needed in a time of 
shrinking state education departments (Massell, 1998, Shields & Knapp, 1997). At the same time, 
states, under standards-based reforms, were attempting to prepare and deliver an arsenal of new 
policy instruments such as curriculum standards and state -wide assessments (Spillane, 1999). 

Fullan (1996) had anticipated problems of fragmentation and overload in systemic change 
efforts and recommended purposeful, structured networking of schools as a means to deal with this 
barrier. Massell (1998) identified three kinds of networks intended to develop the local capacity for 
reform: one that focuses on improving knowledge and skills of participating individuals and 
organizations, one that deploys teachers and other experts to assist local practitioners, and one that 
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develops and distributes products. Bol et al. (1998) studying support for teachers in New American 
Schools restructuring models found that collaboration among teachers emerged as the most positive 
support variable. Even earlier, Lieberman, Darling-Hammond, & Zuckerman (1991) found that 
establishing networks provided inspiration and reassurance as well as answers to educational and 
logistic dilemmas facing teachers in reform. Recommendation of the use of networking as a strategy 
to support reform at both individual and organizational levels is now common (Education 
Commission of the States, 1997, Kimmelman etal., 1999, Research Triangle Institute, 1999). 

FRAMING THE WORK 

This paper reports on an evaluation of a consortium that was formed as a network of 
districts participating in a National Science Foundation (NSF) funded statewide systemic initiative. 
Such intentional networks of districts undergoing systemic reform were few in number in 1993 
when the study began. The consortium offered promise as a means to provide the professional 
development, technical assistance, and collegial learning needed for reform to be successful. 
Participating districts each identified a Design Team of up to six members who attended networking 
conferences at least twice a year over the four years of the consortium’s life. The conceptual and 
structural base for the consortium and its evaluation are found in cluster evaluation, an approach to 
evaluation developed primarily under the auspices of the W.K.Kellogg Foundation (Barley, Z.A. & 
Jenness, M., 1993). As implemented, the cluster evaluation of the consortium included development 
and endorsement of a set of common outcomes, participation in twice yearly networking 
conferences, varying funding support, and technical assistance from a variety of sources. In 
addition, each district was required to develop a strategic plan, form a Design Team, and participate 
in evaluation activities including reporting annually on accomplishments of the goals and objectives 
of the plan. Consortium leaders and the districts, using proposal materials submitted by the districts, 
arrived at consensus on a set of 17 common outcomes in four areas: systemic reform, 
underrepresented groups, community involvement, and mathematics and science curriculum and 
instruction. These became the focus of strategic planning and reporting in each district and the 
focus of the technical assistance activities. 

The evaluators (authors of this paper) served as part of the Management Team for the 
statewide systemic initiative of which the consortium was a component. Opportunities were 
provided in networking conferences for evaluation activities such as interviews and focus groups as 
well as structured within or cross-district reflective work as part of the evaluation. 

STUDY AND METHODOLOGY 



Sample 

The state department of education invited districts to submit an application for funds to 
support systemic reform. Preference in funding was given to low performing districts or districts 
with high minority populations. Following a review and selection process, 24 districts were chosen 
to participate in the consortium. Two of the selected applicants were themselves consortia of very 
small rural districts and are not included in this study. The twenty-two districts in the study included 
one large city, 7 mid-size city districts, 3 suburban areas, and 11 rural locations. All were located 
within a single mid-westem state with a state-wide standardized, criterion-referenced assessment 
system including tests in reading, science, mathematics, and writing at three grade levels. Ten of the 
districts were designated as target districts based on test results and percentage of minority students. 
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Target districts received annual grants rtoging from $25,000 to $65,000 for the first three years of 
the program. The remaining districts were designated affiliate districts and participated in all of the 
professional development and technical assistance but did not receive funding other than that 
needed to attend networking conferences and professional development offerings. Half of the 
affiliate districts qualified for target status but had lower ratings on their proposals and were invited 
to participate without funding (target affiliates). The remaining affiliate districts did not qualify for 
target status but had high ratings on their proposals and were also invited to participate as affiliates. 

Data Collection 

Data collection was extensive and constant beginning with district focus groups held on site. 
Each networking conference was an occasion for additional data collection; for example, interviews 
of Design Team members, review of Design Team data developed pre-conference and brought to 
the meeting, and evaluative data generated by reflective exercises. Documents were collected as 
indicated below, site visits made, and teacher surveys conducted. For each district a portfolio was 
created consisting of: 

■ demographic variables such as size and free/reduced lunch, 

■ annual test scores in mathematics and science, 

■ annual district reports of progress for each of four years, 

■ district proposals for funding and continuation of funding, 

■ site visit transcripts (2 site visits per district in the first and third years), 

■ focus group interviews on site and with Design Team members, 

■ District Progress Report (self evaluation in the third year), 

■ teacher survey (end of third year), 

■ district survey and documents including school improvement reports, curriculum 
materials and disaggregated test scores. 

Data collection for the consortium model consisted of a survey of Design Team members 
soliciting assessment of four program aspects (strategic planning. Design Teams, networking 
conferences, and evaluation activities) and with questions about the availability of funding. A 
lessons learned” workshop was held for all participating districts in the final year. 

Data analysis 

The organizing framework for the district analyses presented in this paper were six elements 
of systemic reform: curriculum, parent-community involvement, classroom practice, professional 
development, administration and leadership, and system change. Progress on these six areas defines 
successful reform in this study. In addition test scores were analyzed for the districts across the four 
years. The consortium itself as a model to support systemic reform was studied through analysis of 
the Design Team survey. 

For the district analysis, the research team created a five-point rubric for each of the six areas 
of reform with anchors describing the “old system” scored as a 1 and an “exemplary” system scored 
as a 5. (See Appendix 1.) Researchers reviewed the portfolios with at least two researchers 
reviewing each district. Inter-rater reliability was established through achieving consensus not only 
between the two reviewers but also through presentation and discussion of scores to establish a 
common understanding of the scoring across the four-member team. 
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The analysis of the consortium included descriptive statistics of the survey data and the post 
conference evaluations. 



RESULTS: DISTRICT LEVEL ANALYSIS 
Six elements of systemic reform 

Frequency of ratings and means for each of the six areas of reform are shown in Table 1. 
The average ratings across all 22 districts on the six areas of reform ranged from 2.5 for classroom 
practice to 3.4 for parent-community involvement. Only professional development had no districts 
rated 1. Both Curriculum and Classroom Practice had no districts rated 5. 



Number of districts rated: 



ELEMENT 


Average 


1 


2 


3 


4 


5 


Parent-Community Involvement 


3.4 


2 


1 


9 


6 


4 


Professional Development 


3.2 


0 


8 


4 


7 


3 


Administration & Leadership 


3.2 


3 


5 


4 


4 


6 


System Change 


2.9 


6 


2 


4 


7 


3 


Curriculum 


2.7 


4 


6 


4 


8 


0 


Classroom Practice 


2.5 


2 


9 


9 


2 


0 



Table 1. District ratings on six elements of reform/l=01d System, 5=Exemplary 

Appendix 2 includes a single page for each of the six elements showing the average rating 
across the 22 districts, the frequency for each rating level and a description of the element for the 
districts rated at each level. For Parent-Community Involvement the two districts rated 1 exhibited 
involvement limited in both amount and type for both parents and community members. The kind 
of involvement was occasional and in traditionally organized events such as parent conferences or 
career days. A role more student-engaged and more innovative parent-community programming 
characterize districts that were rated higher than 1 as does strong effort to recruit/engage these 
groups. The four districts rated 5 demonstrated practices such as including parents in inservices and 
organizing community staffed homework stations in outlying communities. 

No district was rated 1 in Professional Development (PD). The three districts rated 5 had 
developed and implemented strategic plans for PD, had sustained PD with follow up and 
measurable outcomes, and had considerable teacher input to planning and implementation. In 
Administration and Leadership three districts were rated 1. All three had experienced considerable 
turmoil with turnover of leadership. The 6 exemplary districts had empowered teacher leaders; the 
administrator was seen as a strong visionary and advocate for reform who had taken specific actions 
in support. In System Change 6 districts were rated 1. Frequently the role and influence of the 
design team affected the likelihood of systemic change occurring. In one district in this group 
urgent student needs precluded reform. Three districts were rated as 5. Mechanisms for effective 
internal and external networking were evidenced as well as careful capacity building among all 
stakeholders. In Curriculum four districts were rated 1. Typically they had unwritten or unused 
curriculum documents with decisions about what to teach left to the individual teacher. No district 
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was rated a 5. Finally in Classroom practice, 2 districts were rated 1. Teachers reported high use of 
traditional teaching practices. There were no districts rated 5. 

District average total scores 

Total scores were tallied for the 22 districts across all 6 elements. The distribution of 
cumulative scores of the six elements was bimodal with 8 districts having a cumulative score in the 
range of 9 to 13 and 14 districts having a cumulative score between 18 and 27. See Figure 1. 




Figure 1. Distribution of district cumulative scores. 



District reform and student achievement 

In order to explore connections between the rated levels of reform on the six elements and 
student achievement, districts were grouped into three categories: low reform, moderate reform, and 
high reform. The 8 districts shown in Figure 1 with overall scores ranging from 9 to 13 were 
grouped as low reform. The 14 districts with overall scores between 18 and 27 were further divided 
into two groups. The high reform group had high scores on all 6 elements and at least one 
exemplary (5) rating. The moderate group had only moderate ratings across all six with no exemplary 
rating or had uneven ratings across elements. 

To assess student achievement changes during the four years of reform, 4 th grade scores 
from the state mathematics test in the first year of reform were compared to 7 th grade (to some 
degree the same students) scores three years later. Scores in this case are the percent of students 
achieving satisfactory status. Table 2 indicates the average percent achievement change for the three 
reform groups. The right hand column compares that gain to the overall state gain (12.7%) for the 
same period. Thus none of the three groups’ average gains equaled the state gain. 
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Reform Status 


Avg. Group Change 


Compared to State 
Change 


Low 


+0.23 


-12.5 


Moderate 


+ 1.03 


-11.7 


High 


+9.50 


-3.2 



Table 2. Change in student achievement by reform status 



District reform and funding status 

A comparison was made between funding status and reform level. Funding status was either 
as a target district with an annual award ranging from $25,000 to $65,000, or as an affiliate district 
with funding only to support attendance at networking conferences. Affiliate districts were also 
divided into target affiliates - those who initially qualified for target status and therefore were 
academically low performing districts - and affiliates, those who did not qualify as targets. As shown 
in Table 3, 27% (3/11) of the target districts, 29% (2/7) of the target/ affiliates, and 50% (2/4) of 
the affiliate districts were rated high on reform. All three funding groups had districts that did not 
demonstrate high reform. 



Funding Type 


Funded 


Not funded 


Reform Type: 


Target 


Target- 

Affiliate 


Affiliate 


Low 


36% 


57% 


0% 


Moderate 


36% 


14% 


50% 


High 


27% 


29% 


50% 



Table 3 Relationship of reform status to funding type 

District reform and demographics 

The relationship between various district demographic statistics and reform status was also 
examined. Reform status did not relate to percent minority enrollment or percent free and reduced 
lunch. Districts with fewer than 5000 students were found across all three reform categories. 
Districts larger than 5000 students were not found in the high reform status group. 
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DISCUSSION: DISTRICT LEVEL ANALYSIS 

The conclusions that can be drawn from this study are tentative at best. This work 
represents an attempt by an evaluation team to take evaluation findings and make some sense of 
them for the field. The study demonstrates all of the frustrations of not being able to rule out 
alternative explanations in the absence of comparison data. Thus, the reader is cautioned that the 
conclusions that follow must be understood in the light of those limitations. 

The majority of the districts (18/22) were drawn from the group of lowest performing 
districts in the state. Viewed at the end of four years in six areas of reform taken from the literature 
and using those same sources for criteria of accomplishment, 56% (10/18) of these districts were 
able to achieve moderate or high reform status. Those who achieved high reform had higher 
student achievement gains than those who did not. But even then they did not close the gap 
compared to the overall state gain. In this case, an important follow up study will be to revisit these 
districts in three to five years to see if reform has been sustained and student achievement gains 
maintained. Classroom practice was rated lowest of the six elements and unless change occurs there 
student achievement gains are less likely. 

A second area of note is that funding did not play a clear role in accomplishing reform. 
About the same number of funded low performing districts were judged high reform status as the 
non-funded. But more of the non-funded group, which did receive PD and technical assistance, did 
not demonstrate reform. For both funded and unfunded districts, some districts were rated low at 
the end of the four years. None of the affiliate districts were rated low at the end of four years. 

RESULTS: CONSORTIUM LEVEL ANALYSIS 
Analysis/findings from the survey 

A survey of Design Team members conducted at the end of the four years covered five 
elements of the consortium model: 1) required strategic planning framework, 2) biannual technical 
assistance and networking conferences, 3) feedback of district evaluation results, 4) use of a design 
team including community members, and 5) annual funding. Nearly 75% of the respondents 
indicated that their strategic plans were active documents often referred to and revised as necessary. 
They were unanimous in the opinion that the strategic plan helped them focus their work around 
goals and provided away to assess progress and accomplishments. Design team members also 
agreed (96%) that to be effective, strategic planning needed to be integrated with school 
improvement planning (separately required by the state). Only 14% thought strategic plans were 
time consuming efforts with a low yield. Respondents also rated the networking conferences highly. 
The opportunity for networking among colleagues was cited most frequently (85%) as an important 
benefit. Design Team working time at conferences and reflective thinking about the larger issues of 
reform were second and third. Of the various methods evaluators used to provide feedback to 
districts, a midpoint review process in which districts presented their progress and planning for the 
remaining period to a panel of peers - they served on each others panels - was rated by 98% as 
adding value to their thinking and work. The design team structure itself was thought to be an 
important element in implementing reform (98%). The importance of top leadership serving on the 
design team was also cited (75% strongly agreed). Finally 90% of the respondents felt outside 
funding is essential for reform and that it needs to be substantial and continuing. 
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Lessons learned: A group exercise 

In addition to the data described above, findings were also derived from a “lessons learned” 
workshop session conducted at a final conference. The result of a careful process to describe and 
arrive at consensus about the reform process resulted in 11 broad lessons. 

Lesson 1: Two key elements that help establish a context that is ready for reform are vision 
and climate. 

Lesson 2: Each district must identify and use local reasons as an impetus for change. 

Lesson 3: There must be leadership from both top administration and from the classroom. 

Lesson 4: Managing the change process requires a dynamic strategic plan. 

Lesson 5: School resources need to be managed effectively in order for systemic reform to 
be possible. 

Lesson 6: Unique local solutions or ideas for getting external resources — not just money — 
exist and should be sought out. 

Lesson 7: Collaboration with local, state, and national entities reduce the burden on local 
resources. 

Lesson 8: Aligning local curriculum to state and national standards and to high stakes 
assessments is a necessary step. 

Lesson 9: A plan for professional development of teachers and administrators is essential. 

Lesson 10: A process that communicates with and engages stakeholders in the reform 
process requires a plan and emphasis. 

Lesson 1 1 : Reform is a set of processes — not a product. 

UNSCIENTIFIC CONCLUSIONS: CONSORTIA AS A MEANS TO SUPPORT 

REFORM 

The plan to develop a consortium of the districts engaged in this state systemic initiative 
came in part from the evaluator’s previous experience with two cluster evaluations — consortia of 
districts funded by the W.K. Kellogg Foundation - and in part as a result of the NSF review panel’s 
concern that accomplishing reform in low performing districts had not been successful through a 
mode of grantmaking with occasional monitoring. These districts certainly received more direct 
assistance than districts in most other state programs at the time, had more specific requirements for 
changing district practice, and had opportunities to work across districts at various levels to gain 
practical ideas and moral encouragement. Their reflections on the experience suggest a consortium 
model that includes requirements for a design team with key administrative leadership on the team, a 
strategic planning process, PD and technical assistance, and opportunities for networking. While 
districts perceive that outside (additional) funding is necessary, given limited state resources the 
consortia may be of greater use in the long run. 
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